Section: New Results

Matching and 3D tracking

Pose initialization

Automating the camera pose initialization is still a problem in non instrumented environments. Difficulties originate in the possibly large viewpoint changes and lighting variations between the data stored in the model and the current view. One year ago, we began to investigate the use of viewpoint simulation techniques for re-localization within P. Rolin's PhD thesis. We especially consider challenging situations where the current view is distant from the image sequence used for model construction. We here consider scene models built from image sequence using Structure from Motion techniques. A point is then represented by its 3D coordinates and small image patches arising from the images where the point is detected.The underlying idea is to enrich 3D points by descriptors generated from virtual viewpoints chosen away from the learning sequence. For each 3D point of the model, a local image patch is generated from a set of virtual viewpoints, taking into account the local 3D normal and the images of the learning sequence. View synthesis is performed with an affine or an homography model. Though one possible shortcoming of simulation is to generate too many incorrect patches at discontinuities in the scene and thus to degrade the matching step, our preliminary results are very promising [25] and show a noticeable increase of the inlier ratio in the matching stage and an improved stability of the computed pose, especially when homography models are considered. We exhibit many examples where our method successfully computes the camera pose whereas the traditional methods fail.

Current investigations are about the development of scalable solutions for pose computation in large environments with several leverage actions in view. Designing efficient probabilistic techniques for matching and defining strategies based on the geometry of the scene for choosing a reduced set of virtual views are lines of research under investigation for jointly limiting the redundancy and improving the performance of the matching.

Tracking 3D deformable objets

3D augmentation of deformable objects is a challenging problem with many potential applications in computer graphics, augmented reality and medical imaging. Most existing approaches are dedicated to surface augmentation and are based on the inextensibility constraint, for sheet-like materials, or on the use of a model built from representative samples. However, few of them consider in-depth augmentation which is of utmost importance for medical applications. Since the beginning of N. Haouchine's PhD thesis, we have addressed several important limitations that currently hinder the use of augmented reality in the clinical routine of minimally invasive procedures. In collaboration with the SHACRA team, our main contribution is the design and the validation of an augmented reality framework based on a mechanical model of the organ and guided by features extracted and tracked on the video at the surface of the organ [2] . Specific models which best suit the considered organs, such as a vascularized model of the liver, have been introduced in this framework. During this year, we have first performed quantitative evaluation of the method [17] . Promising results were obtained through in-vivo experimentation on a human liver and ex-vivo validation on a porcine liver. In this latter case, artificial tumors were introduced in the liver, thus allowing a quantitative evaluation of the error between the predicted and the actual tumor. These experiments show that localization errors were less than 6mm, and thus below the safety margin required by surgery. To our knowledge, we were the first to produce such evaluation for deformable objects. This work has been extended to augment highly elastic objects in a monocular context [16] , whereas previous works were guided by 3D features obtained with a stereo-endoscope. The only parameter involved in the method is the Young’s modulus but we show in experiments that a rough estimate of the Young’s modulus is sufficient to obtain a good reconstruction. Experiments on computer-generated and real data have shown the effectiveness of the approach. The method is currently restricted to the orthographic projection and its extension to full projective geometry is under investigation.

A bio-mechanical model-based approach has also been considered in the context of tongue tracking in ultrasound images with a view to produce an augmented head for language learning. A crucial issue is the robustness of the tracking due to the strong speckle noise in ultrasound (US) data. Here, a small number of points are used to guide the model. Selection of feature points is based on the uncertainty associated to the tracked points and on spatial constraints. This model has proven to be especially efficient in the case of non uniform and fast movements [19] .

Use of AR in educational sciences

In collaboration with the Ecole supérieure du professorat et de l'éducation and the PErSEUs laboratory at Université de Lorraine, we designed an inquiry-based AR learning environment (AIBLE) for teaching and learning astronomy in primary school (children of 8-11 years old). The novelty of this environment is the combination of Inquiry Based Sciences Education principles and didactics principles (here of astronomy) with AR capabilities. In this context, a GPL-licensed software called AIBLE-AstroAR has been developed based on the ARToolkit library. This software basically consists of a tangible user interface, which allows the children to move virtual celestial objects “as for real” and investigate in order to find origins of Moon phases evolution, alternation of day and night, seasons and Moon/Sun eclipses.

Last year, a study has been carried out to compare AIBLE with a physical model traditionally used in primary school. This study indicated that AIBLE significantly enhances learning compared to classical support. During this year, we performed further investigation with a larger panel of children to assess which characteristics of the environment facilitate learning [14] . Analyses of the marker positions as moved by the children indicated that AIBLE really facilitates heuristic investigation, which fosters consciousness of the origin of astronomical phenomena. This work provides new opportunities for teachers to identify solving problem strategies initiated by learners. These results also contribute to the understanding of the ways through which AR can be used in formal teaching curricula in K-12 schools.